Method and apparatus for QP modulation based on perceptual models for picture encoding

ABSTRACT

A method for encoding a picture is disclosed. The method generally includes the steps of (A) generating at least one respective macroblock statistic from each of a plurality of macroblocks in the picture, (B) generating at least one global statistic from the picture and (C) generating a respective macroblock quantization parameter for each of the macroblocks based on both (i) the at least one respective macroblock statistic and (ii) said at least one global statistic.

CROSS REFERENCE TO RELATED PATENT APPLICATIONS

The present application is a continuation of and claims priority under35 U.S.C. §120 to U.S. application Ser. No. 11/332,025, entitled “Methodand Apparatus for QP Modulation Based On Perceptual Model for PictureEncoding,” filed by Guy Cote on Jan. 16, 2006 now U.S. Pat. No.8,135,062, which is hereby incorporated by reference in its entirety.

FIELD OF THE INVENTION

The present invention relates to picture processing generally and, moreparticularly, to a method and/or architecture for quantization parametermodulation based on perceptual models for picture encoding.

BACKGROUND OF THE INVENTION

Sensitivity of the human eye is not constant under all conditions. Inparticular, contrast variations within different area types in a pictureare commonly perceived differently. Therefore, encoding areas of thepicture where the human eye is less sensitive using the same number ofbits as areas of the picture where the human eye is more sensitive isinefficient.

SUMMARY OF THE INVENTION

The present invention concerns a method for encoding a picture. Themethod generally comprises the steps of (A) generating at least onerespective macroblock statistic from each of a plurality of macroblocksin the picture, (B) generating at least one global statistic from thepicture and (C) generating a respective macroblock quantizationparameter for each of the macroblocks based on both (i) the at least onerespective macroblock statistic and (ii) said at least one globalstatistic.

The objects, features and advantages of the present invention includeproviding a method and/or architecture for quantization parametermodulation based on perceptual models for picture encoding that may (i)increase encoding efficiency compared with conventional approaches, (ii)account for variations in the Human Visual System and/or (iii) adapt toambient (DC) luminance levels in a picture.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects, features and advantages of the presentinvention will be apparent from the following detailed description andthe appended claims and drawings in which:

FIG. 1 is a block diagram of a system shown in accordance with apreferred embodiment of the present invention;

FIG. 2 is a detailed block diagram of a portion of a picturepre-processing module;

FIG. 3 is a detailed block diagram of a portion of an encoder module;

FIG. 4 is a flow diagram for an example method of encoding a picture;

FIG. 5 is a diagram of a contrast threshold at different luminancevalues; and

FIG. 6 is a diagram of an example DC modulation curve.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention generally concerns a quantization parameter (QP)modulation technique using macroblock statistics and picture statisticsgathered from picture preprocessing operations. The macroblockstatistics may comprise (i) a macroblock luminance (luma) motion averagevalue, (ii) a macroblock luma average (DC) value, (iii) a macroblockluma high-frequency average value, (iv) a macroblock luma spatial edgestrength average value and/or (v) a macroblock luma temporal edgestrength average value. The global picture (or global image) statisticsgenerally concern corresponding global picture averages (normalized permacroblock) comprising (i) a global picture luma motion average value,(ii) a global picture luma DC average value, (iii) a global picture lumahigh-frequency average value, (iv) a global picture luma spatial edgestrength average value and/or (v) a global picture luma temporal edgestrength average value. Based on the macroblock statistics and theglobal statistics, a respective quantization parameter for eachmacroblock may be modulated to take advantage of the characteristics ofa Human Visual System (HVS).

Referring to FIG. 1, a block diagram of a system 100 is shown inaccordance with a preferred embodiment of the present invention. Thesystem (or circuit) 100 may be referred to as an encoding system. Theencoding system 100 generally comprises a circuit (or block) 102, acircuit (or block) 104, a circuit (or block) 106 and a circuit (orblock) 108. An input signal (e.g., IMAGE_IN) may be received by thecircuit 102. The circuit 102 may present a signal (e.g., DIN) to thecircuit 104. The circuit 104 may present an intermediate signal (e.g.,INT) to the circuit 108. A signal (e.g., PRE_INFO) may also be presentedfrom the circuit 104 to the circuit 108. A feedback signal (e.g., FB)may be presented from the circuit 108 back to the circuit 104. Thecircuit 108 may present another intermediate signal (e.g., INT′) to thecircuit 106. A signal (e.g., PRE_INFO′) may also be presented from thecircuit 108 to the circuit 106. An output signal (e.g., TS) may begenerated and presented by the circuit 106.

The signal IMAGE_IN may be implemented as either an analog picturesignal or a digital picture signal. When the signal IMAGE_IN is in thedigital form (e.g., DIN=IMAGE_IN), the circuit 102 may be excluded fromthe encoding system 100.

The signal TS may be implemented as video transport signal (or stream).In some embodiments, the transport signal TS (e.g., video bitstream) maybe compliant with an H.264/AVC standard. The H.264/AVC standard ispublished by Joint Video Team (JVT) of the International Organizationfor Standardization/International Electrotechnical Commission (ISO/IEC)Moving Picture Experts Group (MPEG) and the Video Coding Expert Group(VCEG) of the International Telecommunications Union-Telecommunications(ITU-T) Standardization Sector, Geneva, Switzerland. In otherembodiments, the transport signal TS may be compliant with an H.263standard, an H.262 standard or an H.261 standard, also published by theITU-T. In still other embodiments, the transport signal TS may becompliant with an MPEG-2 standard or an MPEG-4 standard, published bythe ISO/IEC MPEG, Geneva, Switzerland. In still other embodiments, thetransport signal TS (e.g., image files) may be compliant with the JPEG,a Tagged Image File Format (TIFF), a Graphics Interchange Format (GIF)and a Portable Network Graphics (PNG) standards. Other image codingstandards, video coding standards and proprietary formats may beimplemented to meet the criteria of a particular application.

The circuit 102 may be referred to as an analog-to-digital (A/D) module.The A/D module 102 may be operational to digitize the input signalIMAGE_IN to generate the signal DIN, where the input signal IMAGE_IN isimplemented as an analog signal. A content of the signal IMAGE_IN may beeither a sequence of video pictures (e.g., frames and/or fields) or astill image. The signal DIN may carry a digital representation of eachpicture in the signal IMAGE_IN.

The circuit 104 may be referred to as a picture pre-processing (PPP)module. The PPP module 104 may be operational to perform pre-encodingprocessing on the pictures in the signal DIN to generate theintermediate signal INT. Processing may include, but is not limited to,spatial processing, color processing, temporal processing, luminanceedge detection, color edge detection, macroblock statistic generation,picture statistic generation and the like for better compression andimage quality results after encoding. The signal INT may carry thepost-processed versions of the pictures received in the signal DIN. ThePPP module 104 may also generate statistics gathered from (i) thepictures and (ii) multiple macroblocks within the pictures. Thestatistics data may be presented in the signal PRE_INFO for temporarystorage in the circuit 108.

The circuit 108 may be referred to as a memory module. The memory module108 may be configured as a buffer for temporarily storing the picturesreceived in the signal INT and the statistic data received in the signalPRE_INFO. The pictures may be read from the memory module 108 in thesignal INT′. The statistics may be read from the memory module 108 inthe signal PRE_INFO′. A portion of the picture data may also be readfrom the memory module 108 in the signal FB. The feedback information inthe signal FB may be used by the PPP module 104 to aid in the processingof subsequent pictures (e.g., temporal processing).

The circuit 106 may be referred to as an encoder module. The encodermodule 106 may be operational to encoding the pictures received in thesignal INT′. The encoded pictures may be presented in the signal TS. Theencoding may result in the signal TS in compliance with at least one ofthe H.264, H.263, H.262, H.261, MPEG-2 and MPEG-4 video standards, theJPEG, TIFF, GIF, PNG image standards and other proprietary formats.

The encoding system 100 may exploit the characteristics of the HumanVisual System (HVS) (e.g., how humans perceive visual information) toachieve a distribution of quantization parameters (e.g., QP) for eachmacroblock in each picture. First, the HVS is less sensitive to highfrequency areas, so a larger quantizer step size may be used in the highfrequency areas. Second, the HVS is less sensitive to high motion areas,so a larger quantizer step size may be used in the high motion areas.Third, the HVS is less sensitive to areas where a DC value is away fromeither (i) the picture DC or (ii) a mid-level value (e.g., a digitalvalue of 128 in a range of 0 to 255), so a larger quantizer step sizemay be used in such areas. Fourth, the HVS is more sensitive to areasthat have strong edges, so a smaller quantizer step size may be used inareas of spatial edges and/or temporal edges. The above factors may becombined to modulate the quantizer step size for every macroblock.

Referring to FIG. 2, a detailed block diagram of a portion of the PPPmodule 104 is shown. The PPP module 104 generally comprises a circuit(or block) 110 and a circuit (or block) 112. The circuit 110 may receivethe digital signal DIN and the feedback signal FB. An internal signal(e.g., SP) may be generated by the circuit 110 and presented to thecircuit 112. The circuit 112 may generate the intermediate signal INTand the signal PRE_INFO written to the memory module 108.

The circuit 110 may be referred to as a spatial processing module. Thespatial processing module 110 may be operational to provide spatialadjustments to the pictures received in the signal DIN to generate theinternal signal SP. The adjustments may include, but are not limited to,cropping, scaling, noise reduction and the like. The spatial processingmodule 110 may also provide advanced information to the encoding module106 by means of statistics gathering. The encoding module 106 generallyuses the advanced information to make better decisions regarding ratecontrol and mode selection when encoding the pictures in the signalINT′.

The circuit 112 may be referred to as a statistics generator module. Thestatistics generator module 112 may be operational to generatestatistics for the pictures at both a macroblock level and at a picturelevel. The statistics may be written to the memory module 108 via thesignal PRE_INFO. Macroblock level statistics generally include (i) aluma motion average value (e.g., MB_(MOT)), (ii) a luma DC value (e.g.,MB_(DC)) that may represent an average luma value among all of thepixels in the macroblock, (iii) a luma high-frequency average value(e.g., MB_(HF)), (iv) a luma spatial edge strength value (e.g., MB_(SE))and/or (v) a luma temporal (motion) edge strength average value (e.g.,MB_(TE)). The luma motion average value MB_(MOT), the luma DC valueMB_(DC), the luma high-frequency value MB_(HF), the luma spatial edgestrength average value MB_(SE) and the luma temporal edge strengthaverage value MB_(TE) may be stored in respective data words (e.g., datawords MbYMotAvg, MbYAvg, MbYHighFreqAvg, MbYEdgeAvg and MbYMotEdgeAvg).

Picture (or global) level statistics generally comprise (i) a globalpicture luma motion average value (e.g., GP_(MOT)), (ii) a globalpicture luma DC value (e.g., GP_(DC)), (iii) a global picture lumahigh-frequency average value (e.g., GP_(HF)), (iv) a global picturespatial edge strength average value (e.g., GP_(SE)) and/or (v) a globalpicture temporal edge strength average value (e.g., GP_(TE)). The globalluma motion average value GP_(MOT), the global luma DC value GP_(DC),the global luma high-frequency average value GP_(HF), the global lumaspatial edge strength average value GP_(SE) and the global luma temporal(motion) edge strength average value GP_(TE) may be stored in respectivedata words (e.g., data words GPictYMot, GPictYSum, GPictYHighFreq,GPictYEdge and GPictMotEdge).

Referring to FIG. 3, a detailed block diagram of a portion of theencoder module 106 is shown. The encoder module 106 generally comprisesan optional circuit (or block) 114, a circuit (or block) 116, a circuit(or block) 118, a circuit (or block) 120 and a circuit (or block) 122.The circuit 114 may receive the intermediate signal INT′ from the memorymodule 108. A signal (e.g., MC) may also be received by the circuit 114.The circuit 114 may present a signal (e.g., RES) to the circuit 116. Thecircuit 116 may present a signal (e.g., CT) to the circuit 118. Thecircuit 118 may present a signal (e.g., QT) to the circuit 120. Thecircuit 120 may present the transport signal TS. A signal (e.g., PAR)may be presented from the circuit 122 to the circuit 118.

The signal MC may carry motion compensated data in embodiments where thesignal INT′ carries a video sequence of pictures. The circuit 114 may beoperational to generate the signal RES as difference between theoriginal picture in the signal INT′ and the motion compensation data inthe signal MC. The difference between the original picture and themotion compensation data may be referred to as residual data.

The circuit 116 may be referred to as a discrete cosine transform (DCT)module. The DCT module 116 may be operational to perform a discretecosine transform on the residual data. The transformed data may bepresented in the signal CT to the circuit 118.

The circuit 118 may be referred to as a quantizer module. The quantizermodule 118 may be operational to quantize the residual data received inthe signal CT based on quantization parameters received in the signalPAR. The quantized data may be presented in the signal QT to the circuit120.

The circuit 120 may be referred to as an entropy encoder module. Theentropy encoder module 120 may be operational to entropy encode the datain the signal QT to generate the transport stream TS.

The circuit 122 may be referred to as a controller module. Thecontroller module 122 may be operational to control the overall encodingprocess of the encoder module 106. In particular, the controller module122 may generate a respective quantization parameter in the signal PARfor each macroblock being encoded based on the statistical informationreceived in the signal PRE_INFO′. The quantization parameters may beprovided to the quantization module 118 in the signal PAR.

Referring to FIG. 4, a flow diagram for an example method 130 ofencoding a picture is shown. The method (or process) 130 generallycomprises a step (or block) 132, a step (or block) 134, a step (orblock) 136, a step (or block) 138, a step (or block) 139, a step (orblock) 140, a step (or block) 142 and a step (or block) 144.

The method 100 generally begins with the A/D module 102 converting theanalog picture data within the signal IMAGE_IN into digital form in thestep 132. The spatial processing module 110 may then perform thepre-processing on the digitized pictures in the step 134. The macroblockstatistics are generally calculated by the PPP module 110 and/or thestatistics generator module 112 in the step 136. At the step 138, thestatistics generator module 112 may calculate the global statistics foreach picture.

In the step 139, the controller module 122 may calculate modulationfactors for each macroblock. The modulation factors may be based one ormore of the local statistics for the respective macroblock and one ormore global statistics for the picture in which the respectivemacroblock resides. In some embodiments, calculations of the modulationfactors may depend on whether the global statistics are greater than orless than the local statistics for the respective macroblocks.

Based on (i) the macroblock luma motion average value MB_(MOT), obtainedfrom the MbYMotAvg field of the data word generated by the PPP module104 and (ii) the macroblock average over the picture value GP_(MOT),obtained by (a) averaging MB_(MOT) over the picture, (b) reading fromthe GPictYMot register or (c) setting to a predetermined fixed value,the controller module 122 may generate a motion modulation factor forluma motion (e.g., α_(MOT)) per equation 1 as follows:

$\begin{matrix}\begin{matrix}{\alpha_{MOT} = {{\frac{\left( {{a \times {MB}_{MOT}} + {b \times {GP}_{MOT}}} \right)}{\left( {{b \times {MB}_{MOT}} + {a \times {GP}_{MOT}}} \right)}\mspace{14mu}{if}\mspace{14mu}{MB}_{MOT}} > {GP}_{MOT}}} \\{= {{\frac{\left( {{c \times {MB}_{MOT}} + {d \times {GP}_{MOT}}} \right)}{\left( {{c \times {MB}_{MOT}} + {d \times {GP}_{MOT}}} \right)}\mspace{14mu}{if}\mspace{14mu}{MB}_{MOT}} \leq {GP}_{MOT}}}\end{matrix} & {{Eq}.\mspace{14mu}(1)}\end{matrix}$where constant pairs (i) “a” and “b” and (ii) “c” and “d” may be used torepresent one or two ratios of the motion modulation factor. In general,the constants a and b may be set such that a motion modulation factorgreater than 1 may be used for macroblocks with motion greater than thepicture average. The constants c and d may be set such that a motionmodulation factor smaller than 1 may be used for macroblocks with motionlower than the picture average.

In order to achieve the above motion modulation factors, (i) the valueof the constant a should be greater than the value of the constant b and(ii) the value of constant c should be greater than the value ofconstant d. In some embodiments, the values of a and b may be set thesame as the values of c and d, respectively. In other embodiments, thevalues of a and b may be set differently than the values of c and d. Forexample, values of a=c=2 and b=d=1 generally represent a motionmodulation factor in a range of [0.5, 2]. As a result, a lower motionmacroblock with respect to the picture average may have a modulationfactor smaller than 1. A higher motion macroblock with respect to theaverage may have a modulation factor greater than 1. Other examplevalues for the constants may include, but are not limited to, (i) a=c=3and b=d=2 for a motion modulation factor in the range of [0.66, 1.5],(ii) a=c=4 and b=d=3 for a motion modulation factor in the range of[0.75, 1.33], etc. In some embodiments, a small motion modulation factor(e.g., a=c=4 and b=d=3) may result in an optimal modulation range.

Based on (i) the macroblock luma DC value MB_(DC), obtained from theMbYAvg field of the data word and (ii) the macroblock average over thepicture value GP_(DC), obtained by (a) averaging MB_(DC) over thepicture, (b) reading from the GPictYSum register or (c) setting to apredetermined fixed value, a DC modulation factor for luma DC (e.g.,α_(DC)) may be calculated by the controller module 122 per equation 2 asfollows:

$\begin{matrix}\begin{matrix}{\alpha_{DC} = {{\frac{\left( {{e \times {MB}_{DC}} + {f \times {GP}_{DC}}} \right)}{\left( {{f \times {MB}_{DC}} + {e \times {GP}_{DC}}} \right)}\mspace{14mu}{if}\mspace{14mu}{MB}_{DC}} > {GP}_{DC}}} \\{= {{\frac{\left( {{h \times {MB}_{DC}} + {g \times {GP}_{DC}}} \right)}{\left( {{g \times {MB}_{DC}} + {h \times {GP}_{DC}}} \right)}\mspace{14mu}{if}\mspace{14mu}{MB}_{DC}} \leq {GP}_{DC}}}\end{matrix} & {{Eq}.\mspace{14mu}(2)}\end{matrix}$where constant pairs (i) “e” and “f” and (ii) “g” and “h” may beconstants used to represent one or two ratios of the DC modulationfactor. A DC modulation factor greater than 1 may be used formacroblocks with DC values away from (e.g., greater than or less than)the average value GP_(DC).

In order to achieve the above DC modulation factor, (i) the constant eshould be greater than the constant f and (ii) the constant g should begreater than the constant h. In some embodiments, the values of e and fmay be set the same as the values of g and h, respectively. In otherembodiments, the values of e and f may be set differently than thevalues of g and h. For example, values of e=g=2 and f=h=1 generallyrepresent a modulation factor in a range of [1, 2], where a lower DCmacroblock or a greater DC macroblock with respect to the average mayhave a higher modulation factor. In some embodiments, a small DCmodulation factor (e.g., e=g=5 and f=h=4) may result in an optimalmodulation range. The DC modulation factor will generally increase themacroblock step size because the DC modulation factor may always begreater than 1. A rate control calculated by the controller module 122may compensate over time for the increased macroblock step size bydecreasing a picture-level quantization parameter.

Based on (i) the macroblock luma high frequency average value MB_(HF),obtained from the MbYHighFreqAvg field of the data word and (ii) themacroblock high frequency average over the picture GP_(HF), obtained by(a) averaging MB_(HF) over the picture, (b) read from the GPictYHighFreqregister or (c) set to a predetermined fixed value, a high-frequencymodulation factor for a luma high-frequency (e.g., α_(HF)) may becalculated by the controller module 122 per equation 3 as follows:

$\begin{matrix}\begin{matrix}{\alpha_{HF} = {{\frac{\left( {{j \times {MB}_{HF}} + {k \times {GP}_{HF}}} \right)}{\left( {{k \times {MB}_{HF}} + {j \times {GP}_{HF}}} \right)}\mspace{14mu}{if}\mspace{14mu}{MB}_{HF}} > {GP}_{HF}}} \\{= {{\frac{\left( {{m \times {MB}_{HF}} + {p \times {GP}_{HF}}} \right)}{\left( {{p \times {MB}_{HF}} + {m \times {GP}_{HF}}} \right)}\mspace{14mu}{if}\mspace{14mu}{MB}_{HF}} \leq {GP}_{HF}}}\end{matrix} & {{Eq}.\mspace{14mu}(3)}\end{matrix}$where constant pairs (i) “j” and “k” and (ii) “m” and “p” may representone or two ratios of the high-frequency modulation factor. The constantsj and k may be set such that a high-frequency modulation factor greaterthan 1 may be used for macroblocks with a high-frequency value greaterthan the picture average value. The constants m and p may be set suchthat a high-frequency modulation factor smaller than 1 may be used formacroblocks with a high-frequency value lower than the picture averagevalue.

In order to achieve the above high-frequency modulation factor, (i) theconstant j is generally greater than the constant k and (ii) theconstant m is generally greater than the constant p. In someembodiments, the values of j and k may be set the same as the values ofm and p, respectively. In other embodiments, the values of j and k maybe set differently than the values of m and p. In some embodiments, alarge modulation factor (e.g., j=m=2 and k=p=1), corresponding to ahigh-frequency modulation factor in a range of [0.5, 2], may provide anoptimal modulation range.

Based on (i) the macroblock spatial edge strength average value MB_(SE),obtained from the MbYEdgeAvg field of the data word and (ii) the globalluma spatial edge strength average value GP_(SE), obtained by (a)averaging MB_(SE) over the picture, (b) read from the GPictYEdgeregister or (c) set to a predetermined fixed value, a spatial edgemodulation factor for a luma edge (e.g., α_(SE)) may be calculated bythe controller module 122 per equation 4 as follows:

$\begin{matrix}\begin{matrix}{\alpha_{SE} = {{\frac{\left( {{q \times {MB}_{SE}} + {r \times {GP}_{SE}}} \right)}{\left( {{r \times {MB}_{SE}} + {q \times {GP}_{SE}}} \right)}\mspace{14mu}{if}\mspace{14mu}{MB}_{SE}} > {GP}_{SE}}} \\{= {{\frac{\left( {{s \times {MB}_{SE}} + {t \times {GP}_{SE}}} \right)}{\left( {{t \times {MB}_{SE}} + {s \times {GP}_{SE}}} \right)}\mspace{14mu}{if}\mspace{14mu}{MB}_{SE}} \leq {GP}_{SE}}}\end{matrix} & {{Eq}.\mspace{14mu}(4)}\end{matrix}$

where constant pairs (i) “q” and “r” and (ii) “s” and “t” may representone or two ratios of the spatial edge modulation factor. The constants qand r may be set such that a spatial edge modulation factor less than 1may be used for macroblocks with a spatial edge strength value greaterthan the picture average value. The constants s and t may be set suchthat a spatial edge modulation factor greater than 1 may be used formacroblocks with a spatial edge strength value lower than the pictureaverage value.

In order to achieve the above spatial edge modulation factor, (i) theconstant q is generally less than the constant r and (ii) the constant sis generally less than the constant t. In some embodiments, the valuesof q and r may be set the same as the values of s and t, respectively.In other embodiments, the values of q and r may be set differently thanthe values of s and t.

Based on (i) the macroblock temporal edge strength average valueMB_(TE), obtained from the MbYMotEdgeAvg field of the data word and (ii)the global luma temporal edge strength average value GP_(TE), obtainedby (a) averaging MB_(TE) over the picture, (b) read from theGPictYMotEdge register or (c) set to a predetermined fixed value, atemporal edge modulation factor for a luma edge (e.g., α_(TE)) may becalculated by the controller module 122 per equation 5 as follows:

$\begin{matrix}\begin{matrix}{\alpha_{TE} = {{\frac{\left( {{v \times {MB}_{TE}} + {w \times {GP}_{TE}}} \right)}{\left( {{w \times {MB}_{TE}} + {v \times {GP}_{TE}}} \right)}\mspace{14mu}{if}\mspace{14mu}{MB}_{TE}} > {GP}_{TE}}} \\{= {{\frac{\left( {{y \times {MB}_{TE}} + {z \times {GP}_{TE}}} \right)}{\left( {{z \times {MB}_{TE}} + {y \times {GP}_{TE}}} \right)}\mspace{14mu}{if}\mspace{14mu}{MB}_{TE}} \leq {GP}_{TE}}}\end{matrix} & {{Eq}.\mspace{14mu}(5)}\end{matrix}$where constant pairs (i) “v” and “w” and (ii) “y” and “z” may representone or two ratios of the temporal edge modulation factor. The constantsv and w may be set such that a temporal edge modulation factor less than1 may be used for macroblocks with a temporal edge strength valuegreater than the picture average value. The constants y and z may be setsuch that a temporal edge modulation factor greater than 1 may be usedfor macroblocks with a temporal edge strength value lower than thepicture average value.

In order to achieve the above temporal edge modulation factor, (i) theconstant v is generally less than the constant w and (ii) the constant yis generally less than the constant z. In some embodiments, the valuesof v and w may be set the same as the values of y and z, respectively.In other embodiments, the values of v and w may be set differently thanthe values of y and z.

The edge detection is generally performed in three dimensions on theluma signal in order to determine static and moving edge transitions.Detection may be performed by parallel Sobel operators. Four possibleedge detection directions may be considered for spatial edges. Fourpossible edge detection directions may be considered for temporal edges.The edge detection directions generally comprise horizontal, vertical,diagonal left and diagonal right directions. For each of the directions,an average edge strength for the macroblock may be computed. A strongestdirection may be identified for both a spatial edge strength and atemporal edge strength. The strength values may be stored in the memory108 for future reference.

Collection of the macroblock statistics may be performed at one or moreamong several different block sizes. For example, 16×16, 16×8, 8×16,8×8, 8×4, 4×8 and/or 4×4 blocks may be initially examined. Depending onthe statistics under consideration, minimum values, maximum values, ormedian values of the smaller blocks (e.g., 8×8 or 4×4) may be used togenerate the statistics for the larger macroblocks (e.g., 16×16).Furthermore, global (or picture) average statistics may be generatedbased on the statistics of the smaller macroblocks and/or the largermacroblocks. For example, a high-frequency statistic for a 16×16macroblock may be set to the highest high-frequency statistic for thesixteen 4×4 blocks of the macroblock. In another example, a spatial edgestatistic for the 16×16 macroblock may be set to the largest spatialedge from the four 8×8 blocks of the macroblock. In still anotherexample, the DC statistic for the 16×16 macroblock may be set to amedian of the smaller blocks within the macroblock. For a motionstatistic of the 16×16 macroblock, an average motion value may becalculated from all of the smaller blocks within the macroblock.

The high-frequency statistics used above are generally an implementationfor measuring an amount of activity present in the macroblocks. In someembodiments, the activity may be determined by measuring other aspectsof the macroblocks. For example, the HVS is less sensitive to hightexture areas. As such, texture or variances within the macroblocks maybe measured. The macroblock quantization parameter may be increased inthe high texture areas.

One or more of the modulation factors α_(MOT), α_(DC), α_(HF), α_(SE)and/or α_(TE) may be optionally low-pass filtered to reduce or minimizefluctuations in the calculated values. Filtering may be performedspatially and/or temporally. The spatial filtering may prevent sharpchanges in the modulation factors over small spatial regions (e.g., aregion of 3×3 blocks to 5×5 blocks). The temporal filtering may preventrapid changes in the modulation factors over a small number of pictures(e.g., 2 pictures to 5 pictures). Other sizes of spatial regions and/ortemporal regions may be implemented to meet the criteria of a particularapplication.

In the step 140, the controller module 122 may calculate a quantizationparameter (or a delta quantization parameter) for each macroblock.Calculation of each of the macroblock quantization parameters may bebased on one or more of the modulation factors. Furthermore, themacroblock quantization parameters may be based on a picturequantization parameter.

The modulation factors described above may be combined by the controllermodule 122 to obtain a quantization parameter (e.g., MB_(QP)) for eachmacroblock. A picture quantization parameter (e.g., PIC_(QP)), generallydetermined by a high level rate control, may be used to modulate eachthe quantization parameter MB_(QP) of each macroblock. First, thepicture quantization parameter PIC_(QP) may be converted to a quantizerstep size (e.g., PIC_(QUANT)). For H.264, the quantizer step sizePIC_(QUANT) may be calculated per equation 6 as follows:PIC _(QUANT)=2^(max(PIC _(QP)−4,0)/6)  Eq. (6)For MPEG-2, the quantizer step size PIC_(QUANT) may be calculated perequation 7 as follows:PIC _(QUANT)=2^PIC _(QP)  Eq. (7)Other linear and/or non-linear quantizer step sizes may be calculatedbased on the criteria of a particular application.

The quantization parameter to quantizer step size conversion (e.g.,equations 6 and 7) may be implemented as a table lookup (e.g., LookupTable 109 in FIG. 1). The table lookup generally allows for a linear ora non-linear transformation of the quantizer step size based on themodulation factors. The modulated quantizer step size for eachmacroblock of the picture may be obtained by multiplying all themodulation factors by the picture quantizer value per equation 8 asfollows:MB _(QUANT)=α_(MOT)×α_(DC)×α_(HF)×α_(SE)×α_(TE) ×PIC _(QUANT)  Eq. (8)

Once the modulated macroblock quantizer value MB_(QUANT) is obtained,the macroblock quantizer value MB_(QUANT) may be converted back to themacroblock quantization parameter value MB_(QP). For H.264, theconversion may be performed per equation 9 as follows:MB _(QP)=6 log₂(α_(MOT)×α_(DC)×α_(HF)×α_(SE)×α_(TE))+PIC _(QP)  Eq. (9)Similar conversions may be performed for other standards and proprietaryformats. The conversion from macroblock quantizer value MB_(QUANT) intothe macroblock quantization parameter value MB_(QP) may be implementedas a reverse table lookup.

In some embodiments a quantization parameter delta value (e.g.,ΔMB_(QP)) may be calculated and used instead of the multiplicativefactor MB_(QP). Use of the quantizer parameter delta value ΔMB_(QP) maybe simpler in that the picture quantizer value PIC_(QUANT) may remainundetermined. As such, a macroblock quantization parameter delta listfor all macroblocks in the picture may be generated strictly from thelocal (macroblock) statistics and the global (picture) statisticsgathered by the PPP module 104. From equations (6) and (8), themacroblock quantization parameter delta value for H.264 may becalculated per equation 10 as follows:ΔMB _(QP)=6 log₂(α_(MOT)×α_(DC)×α_(HF)×α_(SE)×α_(TE))  Eq. (10)Similar calculations may be performed for other standards andproprietary formats.

The macroblock quantization parameters MB_(QP), quantizer step sizesMB_(QUANT) and/or delta quantization parameters ΔMB_(QP) may beoptionally low-pass filtered to reduce or minimize fluctuations in thecalculated values. Filtering may be performed spatially and/ortemporally. The spatial filtering may prevent sharp changes in thequantization step sizes over small spatial regions (e.g., a region of3×3 blocks to 5×5 blocks). The temporal filtering may prevent rapidchanges in the quantization step sizes over a small number of pictures(e.g., 2 pictures to 5 pictures). Other sizes of spatial regions and/ortemporal regions may be implemented to meet the criteria of a particularapplication.

In the step 142, the difference module 114, the DCT module 116, thequantization module 118 and the entropy encoder module 120 may encodethe pictures based in part on the individual macroblock quantizationparameter values (or the macroblock quantization parameter deltavalues). The transport stream TS may be transmitted from the encodermodule 106 and/or stored in a medium in the step 144.

Simulations were performed on the method 100. For simulation purposes,control was provided to (i) enable/disable each modulation factorindividually and (ii) set each modulation factor ratio represented bythe constant pairs.

Sensitivity of the human eye to changes in luminance contrast aregenerally inconsistent over the luminance range. In particular, thehuman eye is most sensitive to changes in contrast in the mid greylevel. Thus, humans appear to be more sensitive to coding artifacts inthe mid-grey level rather than the high and low luminance levels and thequantization step should be modulated accordingly.

In order to measure the contrast threshold, a luminance level of a smallpatch (e.g., 64×64) of pixels located at the center of flat area ofconstant luminance value was varied. The luminance level difference atwhich the patch could be perceived different from the backgroundgenerally represented the contrast threshold for the particularluminance level of the background. The experiment was carried out forthe entire luminance range to measure the contrast threshold at eachluminance level.

Referring to FIG. 5, a diagram of the contrast threshold at differentluminance values is shown. A first curve 146 was derived from theexperiments while a second curve 148 is a conventional just-noticeabledistortion (JND) contrast threshold curve. The experiments generallyshow that the human eye is most sensitive to grey level of 64 (in arange of 0 to 255) while the JND curve 148 suggests that the eye is mostsensitive to a grey value of 128. A possible explanation for thedifferences may be that the conventional contrast thresholds are highlysensitive to the luminance and the ambient light conditions.

The curves 146 and 148 generally show that the human eye is lesssensitive to contrast changes at the lower luminance level than thehigher luminance levels. The contrast sensitivity generally decreasesexponentially moving away from the mid-grey level to the lower luminancelevels. Furthermore, the contrast sensitivity tends to decrease in alinear fashion moving toward the higher luminance values.

The curve 146 suggests that the modulation factor for the luminance maybe modulated in such a way to produce (i) an exponential increase in thevalues of alpha for lower luminance values and (ii) a linear increasefor higher luminance values. The JND curve 148 suggests that the eye ismost sensitive to a luminance value of 128 and thus a constant value of128 may be used for the global luma DC average value GP_(DC) in equation(2).

Referring to FIG. 6, a diagram of an example DC modulation curve 150 isshown. The DC modulation curve 150 may be obtained by fixing the valueof the global luma DC value GP_(DC) to 128 and using values of g=3, h=1for luminance values less than 128 and using values of e=5, f=2 forluminance values greater than 128.

Various encoding were done at different bit-rates to test whether theconstant value of 128 was a better choice than using the global averageof luma in equation (2). From theoretical arguments, a constant value of128 should have been more suitable than the varying global average.Other values, such as 64 and 96 were also examined. The encoding of thepictures performed with the constant value of 128 generally lookedvisually better than using 64 or 96. However, the encoding of thepictures performed using the actual global average of luma typically hadbetter visual quality than using the constant value of 128. As such,equation (2) may be implemented as follows:

1. Use the global average instead of a constant value.

2. In cases where MB_(DC)<GP_(DC), then g=3 and h=1.

3. In cases where MB_(DC)>GP_(DC), then e=5 and f=2.

The function performed by the diagrams of FIGS. 1-3 and the flow diagramof FIG. 4 may be implemented using a conventional general purposedigital computer programmed according to the teachings of the presentspecification, as will be apparent to those skilled in the relevantart(s). Appropriate software coding can readily be prepared by skilledprogrammers based on the teachings of the present disclosure, as willalso be apparent to those skilled in the relevant art(s).

The present invention may also be implemented by the preparation ofASICs, FPGAs, or by interconnecting an appropriate network ofconventional component circuits, as is described herein, modificationsof which will be readily apparent to those skilled in the art(s).

The present invention thus may also include a computer product which maybe a storage medium including instructions which can be used to programa computer to perform a process in accordance with the presentinvention. The storage medium can include, but is not limited to, anytype of disk including floppy disk, optical disk, CD-ROM,magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, Flash memory,magnetic or optical cards, or any type of media suitable for storingelectronic instructions.

While the invention has been particularly shown and described withreference to the preferred embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade without departing from the spirit and scope of the invention.

What is claimed is:
 1. A method for encoding a picture, comprising thesteps of: (A) generating at least one respective macroblock statisticfrom each of a plurality of macroblocks in said picture, the at leastone respective macroblock statistic being selected from a group thatconsists of a luminance motion value, a luminance DC value, a luminancehigh-frequency value, a spatial edge strength value and a temporal edgestrength value; (B) generating at least one global statistic from saidpicture by averaging the at least one respective macroblock statistic ofthe plurality of macroblocks over said picture; and (C) generating arespective modulation factor and a respective macroblock quantizationparameter for each of said macroblocks based on both (i) said at leastone respective macroblock statistic and (ii) said at least one globalstatistic.
 2. The method according to claim 1, wherein each of saidrespective macroblock quantization parameters is further based on apicture quantization parameter.
 3. The method according to claim 1,wherein each of said respective macroblock quantization parameterscomprises a respective delta quantization parameter generated only fromsaid at least one respective macroblock statistic and said at least oneglobal statistic.
 4. The method according to claim 1, said respectivemacroblock quantization parameter is generated based on said respectivemodulation factor.
 5. The method according to claim 1, wherein the stepof generating said respective modulation factor is calculated using (i)a first equation if said at least one respective macroblock statistic isgreater than said at least one global statistic and (ii) a secondequation if said at least one respective macroblock statistic is lessthan said at least one global statistic.
 6. The method according toclaim 1, further comprising the step of: spatially filtering saidrespective modulation factors prior to generating said respectivemacroblock quantization parameters.
 7. The method according to claim 1,further comprising the step of: temporally filtering said respectivemodulation factors prior to generating said respective macroblockquantization parameters.
 8. The method according to claim 1, furthercomprising the step of: spatially filtering said respective macroblockquantization parameters.
 9. The method according to claim 1, furthercomprising the step of: temporally filtering said respective macroblockquantization parameters.
 10. The method according to claim 1, wherein atleast one of said at least one global statistics comprises a fixedvalue.
 11. The method according to claim 1, further comprising the stepof: spatially processing said picture before generating said at leastone respective macroblock statistic.
 12. The method according to claim1, wherein said picture forms a portion of a video stream.
 13. Themethod according to claim 1, wherein said picture comprises a stillimage.
 14. A system comprising: a processor configured to (i) generateat least one respective macroblock statistic from each of a plurality ofmacroblocks in a picture and (ii) generate at least one global statisticfrom said picture, the at least one respective macroblock statisticbeing selected from a group that consists of a luminance motion value, aluminance DC value, a luminance high-frequency value, a spatial edgestrength value and a temporal edge strength value, the at least oneglobal statistic averaging the at least one respective macroblockstatistic of the plurality of macroblocks over said picture; and anencoder configured to generate a respective modulation factor and arespective macroblock quantization parameter for each of saidmacroblocks based on both (i) said at least one respective macroblockstatistic and (ii) said at least one global statistic.
 15. The systemaccording to claim 14, wherein said encoder is further configured togenerate a respective macroblock quantization step size for each of saidmacroblocks based on said respective macroblock quantization parameter.16. The system according to claim 15, further comprising a memoryconfigured to store a lookup table for converting said respectivemacroblock quantization step sizes to said respective macroblockquantization parameters.
 17. The system according to claim 14, whereinsaid at least one respective macroblock statistic comprises a texturevalue.
 18. The system according to claim 14, wherein said at least onerespective macroblock statistic comprises at least one of (i) aluminance motion value, (ii) a luminance DC value, (iii) a luminancehigh-frequency value, (iv) a spatial edge strength value and (v) atemporal edge strength value.
 19. The system according to claim 14,wherein said encoder is further configured to encode said picture intoan output signal compliant with at least one of an InternationalOrganization for Standardization/International ElectrotechnicalCommission 14496-10 standard and an International TelecommunicationUnion-Telecommunications Revised Recommendation H.264.
 20. A systemcomprising: means for generating at least one respective macroblockstatistic from each of a plurality of macroblocks in a picture, the atleast one respective macroblock statistic being selected from a groupthat consists of a luminance motion value, a luminance DC value, aluminance high-frequency value, a spatial edge strength value and atemporal edge strength value; means for generating at least one globalstatistic from said picture by averaging the at least one respectivemacroblock statistic of the plurality of macroblocks over said picture;and means for generating a respective modulation factor and a respectivemacroblock quantization parameter for each of said macroblocks based onboth (i) said at least one respective macroblock statistic and (ii) saidat least one global statistic.